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METHOD AND APAR4TUS FOR CHARACTERIZING 
THE QUALITY OF A NETWORK PATH 

Field 

The invention relates generally to paths of one or more computer 
5 networks, and particularly relates to metrics characterizing the network paths. 

Background 

The use of efficient routing algoritlims, such as the bellman-ford 
algorithm and the open shortest path (OSP) algorithm is highly desirable in 
complex networks. Until today, these algoritlims use metrics that are mostly 

1 0 static, and that are based on very coarse approximations of path performance 
quality (e.g., number of hops, user-defined static costs that are associated to 
each of the links in the network). As the internet becomes more and more 
ubiquitous, metrics that characterize the quality of network applications running 
across network paths become more important! Metrics for voice and video have 

15 been devised in the past; however these metrics are complex and not adapted for 
routing, and instead can be used to report performance at select points in the 
network. 

Summary 

K4etrics that at the same time are (1) additive and (2) characterize the 
20 performance of network applications are highly desirable, as they allow routing 
algorithms and devices that can take into account factors that matter to the users 

of these applications. 

We describe multiple methods and apparatuses for characterizing the 
quality of a network path by means of a metric. This metric characterizes a 
25 plurality of one or more network applications running across a network path. 

The quality characterization characterizes a quality of the same plurality of one 
or more network applications running at one or more end-points of the path. 
This metric is at least a function of a plurality of one or more elementary 
network parameters. The plurality of one or more network parameters include 
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one or more of delay, jitter, loss, currently available bandwidth, and intrinsic 
bandwidth. 

This metric is also additive, in the sense that given a network palli that 
includes a first segment and a second segment, the metric score across the 
network path is equal to the sum of the metric score across the Fu st segment, 
and the metric score across the second segment. 

Additional features, benefits, and embodiments of the present invention 
will become apparent fi-om the detailed description, figures and claims. 


'max 


Brief Description of the Drawings 

Fig . 1 voice traffic: performance degradation {MOS - MOS,„i„y{MOS, 
- MOS,„i„) of some embodiments versus (a) percentage speech clipped and (b) 
RTT. 

Fig. 2 short TCP connections: performance degradation 

(Latency„JLatency) of some embodiments versus (a) the loss rate and (b) RTT. 

Fig. 3 t}'pical file transfer: performance degradation 

{Throiighpui/Throughpul,,,^;) of some embodiments versus (a) the loss rate and 
(b) RTT. 

Fig. 4 MOS contours of some embodiments versus 7?7Tand % speech 

lost. 

Fig. 5 normalized MOS degradation of some embodiments versus (a) 
speech lost and (b) one-way delay with negative exponentials. 

Fig. 6 shows an embodiment of approximating Latency,,,,,! Latency of 
some embodiments versus (a) one-way delay and (2) loss rate with negative 
exponentials. 

Fig. 7 shows an embodiment of approximating Latency,,,,,/ Latency of 
some embodiments versus (a) one-way delay and (2) packet loss with negative 
exponentials. 

Fig. 8 approximating Throughpui/Throughpru„,,_, o( somz embodiments 
versus (a) one-way delay and (2) packet loss with negative exponentials. 
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Fig. 9 approximating Throughpui/Tliroughpul max- of some embodiments 
versus (a) one-way delay and (2) packet loss with negative exponentials. 

Fig. 10 series of mappings involved in the translation of raw 
measurement traces into a web transaction score of some embodiments. 

5" Fig. 1 1 dynamics of a Simple TCP connection of some embodiments. 

Fig. 12 cwnd versus time for a TCP connection over of some 
embodiments (a) a high bandwidth path and (a) a low bandv^adth path. 

Fig. 13 download of a web page which consists of one html file and nine 
images using HTTP/1 .0, with the Keep-Alive option either set or not of some 
1 0 embodiments. 

Fig. 14 download of a w^eb page, which consists of one html file and 
nine images using HTTP/1.1, using either persistent connections, or persistent 
and pipelined connections of some embodiments. 

Fig. 15 latency of a typical web transaction for different file sizes, given 
1 5 various loss rates versus one-way delay, using either HTTP/1 .0 or HTTP/1 . 1 of 
some embodiments. 

Fig. 16 typical web transaction for different file sizes, given various 
round-trip times versus the loss rate, using either HTTP/1 .0 or HTTP/1.1 of 
some embodiments. 

20 Fig. 17 latency of a typical web transaction for different file sizes, given 

various round-trip times versus one-way jitter, using either HTTP/l .0 or 

HTTP/1 .1 of some embodiments. 

Fig. 18 user's impression versus web transaction duration A. Bouch, N. 

Bhatti, and A, J. Kuchinsky, Oiialily is in the eye of the beholder: hdeeiing 
25 . users' requirements for Internet Quality of Service, To be presented at 

CHr2000. The Hague, The Netherlands, April 1-6, 2000, pp. 297-j04.and A. 

Bouch,N. Bhatti, and A. J. Kuchinsky, Integrating User-Perceived Quality into 

Web Server Design, HP Labs Technical Report HPL-2000-3 20000121 of some 

embodiments. 

30 Fig. 19 impermissible rate (%) versus MOS, N. Kitawaki and K. Itoh, 

Pure Delay Effects on Speech Quality in Telecommunications, IEEE Journal of 
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10 


15 


20 


Selected Areas in Communications, Vol. 9, No. 4, May 1991 of some 
embodiments. 

Fig. 20 metric of a typical web transaction for different file sizes, given 
various loss rates versus one-way delay, using either HTTP/1.0 or HTTP/1 . 1 of 
some embodiments. 

Fig. 2 1 metric of a typical web transaction for different file sizes, given 
various round-trip times versus the loss rate, using either HTTP/1 .0 or 
HTTP/] .] of some embodiments. 

Fig. 22 metric of a typical web transaction for different file sizes, given 
various round-trip times versus one-way jitter, using either HTTP/1.0 or 
HTTP/1. 1 of some embodiments. 

Fig. 23 network device of some embodiments deployed in a network. 

Fig. 24 network device of some embodiments deployed in an 
internetwork, at a point of presence between one or more networks. 

Fig. 25 network device of some embodiments used for routing deployed 
in an internetwork. 

Fig. 26 network device of some embodiments used for routing, deployed 
in an internetwork, at a point of presence between one or more networks. 

Fig. 27 shows some possible embodiments with devices that are 
communicating with each other, for example sending and receiving 
measurement packets. 

Fig, 28 shows one specific detailed embodiment with two devices 
where each device is sending and receiving measurement packets as well 'as 
selectmg a subset of paths. 

Fig. 29 shows an embodiment with more than two devices that are 
sending and receiving measurement packets to obtain measurements of 
performance characteristics of paths and to communicate measurements 
statistics about those paths. 
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Detailed Description 

We describe multiple embodiments of a method that translates 
elementary network parameters, wherein the plurality of one or more network 
parameters include one or more of delay, jitter, loss, currently available 

5 bandwidth, and intrinsic bandwidth into a metric that measures, at least in part 
quality characterizations of a same plurality of one or more network 
applications, wherein the quality characterization characterizes a quality of the 
same plurality of one or more network applications user-perceived quality 
metrics. In some embodiments of this invention, this metric captures the user- 

1 0 perceived experience of the performance of an application. This metric is 
additive, in the sense that given: 

- a network path, including a first segment and a second segment; 

- a first metric and the second metric, which are at least in part 
quality characterizations of a same plurality of one or more 

1 5 network applications, wherein the quality characterization 

characterizes a quality of the same plurality of one or more 
network applications running at one or more segment end-points 

- wherein the first metric and the second metric are at least partly a 
function of a same plurality of one or more elementary network 

20 parameters, wherein the plurality of one or more network 

parameters include one or more of delay, jitter, loss, cuiTcntly 
available bandwidth, and intrinsic bandwidth 

- wherein the first metric is at least partly the function of the same 
plurality of elementary network parameters of the first segment, 

25 wherein the one or more segment end points include one or more 

end-points of the first segment 

- wherein the second metric is at least partly the function of the 
same plurality of elementary network parameters of the second 
segment, wherein the one or more segment end points include 

30 one or more end-points of the second segment, 
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adding the first metric and the second metric generates a third metric, wherein 

- the third metric is at least partly the function of the same 
plurality of one or more elementary network parameters of the 
network path, wherein the one or more segment end points 
include one or more end-points of the network path 

- the third metric is a quality characterization of a same plurality 
of one or more applications 

In the process of designing such a metric, we first describe the insight that 
drives the general methodology, as well as the methodology specifics for voice 
video,. TCP traffic, HTTP/1 .0, and HTTP/1 . 1 respectively. Applying the 
described teclinique for voice, video, and data traffic, we derive embodiments of 
such an application specific metric for each type of application independently. 

General methodology 

Let's look at the general shape of performance degrac/alion versus delay 
and loss curves for voice and TCP data traffic, where performance means the 
performance metric that matters for the respective application: MOS score for 
voice traffic, throughput and latency for TCP data traffic. (See Fiss. 1-3.) 

In some embodiments of this invention, these curves are nomialized 
Normalizing the performance curves includes translating them to a 0-1 
performance score, where 1 represents no degradation, whereas 0 represents 
total degradation. Other embodiments can use other scales such as inverted 
scales from 1-0 and/or different scales such as 0-10. For example, in some 
en.bodi.ents of this invention, .„ one embodiment of the voice application, a 
MOS (Mean Opinion Score) of 4 is converted to a metric of 1 for voice (since 4 
.s the maximum that can be attained in a telephone system), whereas a MOS 
score of 1 maps to a metric of 0 (that is, total degradation). On the other hand in 
some embodiments of this invention, for general TCP applications, it is 
assumed that a metric of 1 for short TCP transactions translates into a fast 
rehable response, whereas a metric of 0 represents "infinitely" slow response 
Fmally, .„ some embodiments of this invention, for TCP file transfer a metric 
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of 1 corresponds to a high throughput, while a metric of 0 corresponds to a 
throughput that is close to 0. Note that in the context of TCP, a loss rate. of 0 is a 
physically plausible phenomenon that gives the benchmark performance in 
terms of latency and throughput to which the performance achieved over lossy 

5 paths can be compared to. On the other hand, for some embodiments of this 
invention, a round-trip time of zero is less realistic, and leads in one 
embodiment with TCP traffic to both infinite throughput and zero latency as 
long as the loss rate is less than 100%. Hence, in some embodiments of this 
invention, the round-trip time for which the benchmark latency and throughput 

1 0 measures are computed, RTTq is larger than 0, and can be chosen according to 
the characteristics of the particular system in consideration. In some 
embodiments, the internetwork considered includes national networks spanning 
large geographical areas; in this context, a one-way delay of less than 25 ms is 
uncommon. Hence, in such embodiments, a RTTq = 50 ms can be appropriate. 

1 5 In some embodiments, the choice of RTTq is significant, since different values 
ofRTTo lead to different shapes for the Gur\'es in Figure 2b and Figure 3b, 
which in turn lead to different metric parameters. 

The shape these curves is very similar between many embodiments. In 
some embodiments, the curves corresponding to voice (Fig. I) iiave a shape that 

20 can be approximated by a negative exponential. In some embodiments, the 
curves corresponding to TCP applications (Figs. 2-3) are hyperbolic in p and 
RTT: in some embodiments of this invention, a hyperbolic function can be 
approximated, for a portion of the parameters' ranges with an exponential 
function. The related issues are dealt with later in the appendix, in the section 

25 reserved for TCP applications. 

In some embodiments, it is assumed that these curves can be fitted by 
negative exponential functions in some portion of the parameters' ranges. In 
some embodiments, in both the case of voice and data traffic, it is also assumed 
that one-way delay is half the round-trip time delay, so performance degradation 

30 versus one-way delay curves can be obtained. The same theory can apply to 
embodiments of voice and/or TCP traffic. For simplicity, in the remainder of 
this section, we describe an embodiment in the context of voice traffic. Hence, 
in this embodiment, the following equations estimate performance degradation 
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versus one-way delay and loss, nivd ^^'^^ ^vh respectively, corresponding to 
voice traffic: 
^72,,, =exp(-a,D) 
/;/., ^exp(- /?../) 

5 

Delay-loss MOS contours have not, to the extent of our knowledge, been 
studied extensively in previous subjective studies of voice quality. Hence, in 
some embodiments, assumptions are made in the way a metric niy combining 
m,,o and m,,i can be obtained from both metrics' equations. In one embodiment, 
1 0 one intuitively appealing technique that can be used to combine niyo and can 
be used. Given the equations above, performance is close to perfect when both 
niyo and ///,./ are close to 1 . That is, we have 
{niyo 1 and niy/ my = 1 

1 5 On the other hand, note that if either niyo or /»w are close to 0, then the quality as 
perceived by the user will be bad, i.e., niy must be close to 0. Hence, the second 
relation between niyo, myi and niy ought to be 


20 


{WyD = 0 or Wyi = 0) => niy == 0 

One operator that satisfies both relations above is x. hi this embodiment, we use 
this operator; the resulting metric for voice becomes 

X m,, = exp(^a,D)xexp(-^,./) = exp(-a..D-^,/) 

According to the obtained meiric, equal metric contours in the (D, /) plane are 
straight lines (See Fig. 4.) 


Note that since the exponential function is monotonic, the metric is also 
30 additive, in the sense described above, (which we denote MS in this document) 
is equal to 


25 
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melric = D + —1 


Defining 5v to be equal to (3,yav, we get 


mclric = Da- 5 J 

5 

In this embodiment, the metric has two properties: on one hand, it can be 
simply modified using a single parameter 5,.. (Other embodiments can use more 
than one parameter.) On the other hand, it is additive, in the sense described 
above. This characteristic can be useful in the context of routing; specifically, in 

] 0 some embodiments, this melric can be used for routing, using algorithms such 
as Distance Vector and SPF (Shortest Path First). That is, using such a melric, 
dogleg routing can be easily implemented. In the following, we justify this 
property in the context of this specific embodiment: assume a path includes two 
links, L\ and Li- The one-way delay and loss across linkLi are denoted D\ and 

15 /i, while the delay and loss across link L2 are denoted D2 and I2, respectively. 
We also define meiric\ and metriC2 to be the resulting metrics on link L\ and 
linkio, respectively, while the total metric metric is set to meiric] + metic2 = D\ 
+ D2 + 0,. (/i + h). In this particular example, w^e use the one-w^ay delay equation 

D. =c-/-+4^-, / = L2 

20 Cleariy, D = Di + Do is only an approximation of the delay on the link 

since in some embodiments, jitter is not additive. However, in some 
embodiments, the total jitter across a path is smaller or equal to the sum of jitter 
values for all segments on the path. Hence, in these embodiments, D| + 1)2 is a 
higher bound on the actual delay D over the link. In other words, setfing the 

25 delay for the total path to D\ ^ D2. a 2 hop path is effectively penalized over a 
one hop path. In some embodiments, this is intuitively justifiable, since using 
two hops as opposed to one hop exhibits potentially extra overhead which is not 
taken into account by the delay metric, such as the potential additional delay 
incurred at the node connecting the two hops, and other reliability issues in 

30 terms of routing and connectivity. Hence, in such embodiments, it appears 
reasonable to penalize routes w^ith a higher hop count. 


9 


wo 02/33896 


PCT/USOl/32476 


On the same h'ne of thoiiglit, let's derive the total loss rate on the path. Jn 
some embodiments, the losses on Links Z,, and L2 can be assumed to occur 
independently; in such embodiments, then the total loss can be found using 

!-/ = (! -/|)(l-/2) 

/ = /|+/2-/|/2 

That is, the actual loss on the path is lower than the value /, + k assumed by the 
additive nature of the metric. In fact, in embodiments where the losses on links 
L\ and Li are correlated, the actual loss is generally lower than /, + /, - /lA. 
However, first note that for the range of loss rates of interest to some 
embodiments, (say 1-10%), hh is much smaller than /, or /. (for example, for /, 
- 1 0% and /, = 10%, then /,/, is a mere 1%). Hence, in such embodiments, /,A 
can safely be ignored from the equation for /. Furthermore, in some 
embodiments, the same argument can be set forth, as was iiiade for the 
computation of delay, namely that penalizing the route that has a larger hop 
count is justifiable from a design point of view. 

The arguments described above demonstrate the adequacy and 
practicality of the metric metric = D + 5/ for some embodiments. In tlie sections 
below, the theory described above, which applies to some embodiments of this 
invention, is used in the specific contexts of voice and TCP data traffic, 
respectively, leading to some embodiments of this invention for voice and TCP. 
In some embodiments, it is useful to have a value of 5 that is adequate both for 
voice and TCP. In some embodiments the value of 5 is common for both voice 
and data traffic. 


Metric for voice traffic 

In this section, we appro.ximate the normalized A/C?^ degradation curves 
shown in Fig. 1 with negative e.xponentials. The results are shown in Fi" 5 
Focusing first on Figure 5a, we can see that fitting the curve with an e.Kponcntial 
leads to either an under-estimation of the degradation for low speech loss (that 
.s, less than 4% loss rates) or to an over-estimation of the degradation for h.,h 
loss rates (e.g., 1 0% loss rates and above). In order to pick an appropriate fit 
one must remember that the curve shown in Fig. 5a assumes no error resiiiencv 
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at all. Tliat is, a lost packet is replaced with silence. However, most modem 
voice decoders have features that enable some sort of error resiliency, in such a 
way that the effect of small clips can be significantly mitigated. As a result, one 
might expect that in actual modern voice encoders, the MOS degradation 

5 corresponding to low speech lost is not as high as that shown in Figs. 1 a and 

Fig. 5a. In addition, it is clear that the efficiency of such error-resilient decoders 
gets low^er as the packet loss increases; for loss rates exceeding 10%, no error 
correction can overcome the degradation caused by the amount of information 
lost. Hence, as shown in Fig. 5a, we fit the degradation curves with negative 

1 0 exponentials that under-estimate the degradation for low speech loss (that is, for 
values of / lower than 4%), and matches the degradation for values of/ 
exceeding 6%. We get 

exp(- 25/) </?/,/ <exp(-20/)^ 
with an average of 

15 mv/ = exp(-23/) 

Similarly, we fit negative exponentials to the cur\'es representing MOS 
degradation versus one-way delay D (obtained from the MOS versus RTT 
curves using the simple D = RTTIl relation). (See Figure 5b.) Here too, in some 
embodiments of this invention, a choice has to be made between slightly under- 
20 estimating the loss for low values of delay, and over-estimating the loss for 
higher values of delay. In some embodiments, the curves are derived from 
experiments conducted in N. Kitawaki and K. Itoh, Pure Delay Effects on 
Speech Oiialiiy in Telecommimicuiions, IEEE Journal of Selected Areas in 
Communications, Vol. 9, No.4, May 1991,for different tasks: 

25 

• Task J: Take turns reading random numbers aloud as quickly as possible 

• Task 2: Take turns verifying random numbers as quickly as possible 

• Task 4: Take turns verifying city names as quickly as possible 

• Task 6: Free conversation 

30 In some embodiments of this invention, the experiments that involve 

intense interaction, such as business calls or transaction-related calls are the 
most relevant. For these embodiments, the metric is optimized for tasks that 
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resemble more Tasks 1 and 2 (from N. Kitawaki and K. Itch, Pure Delay Effects 
on Speech OuaJi/yin Telecommunications, IEEE JoumaJ of Selected Areas in 
Communications, Vol. 9, No.4, May 1991, then Tasks 4 and 6. In addition, for 
such embodiments, a round-trip time that is larger than 500ms (that is, one way 
delays larger than 250 ms) is not desirable, as it gives a sense of poor quality 
that is not well reflected by the low performance degradation obtained in N. 
Kitawaki and K. ftoh, Pure Delay Effects on Speech Quality' in 
Telecommunications, IEEE Journal of Selected Areas in Communications. Vol. 
9, No.4, May 1991. Thus, for such embodiments, the chosen voice negative 
exponential curves fit closely the results obtained in N. Kitawaki and K. Itoh, 
Pure Delay Effects on Speech Qualily in Telecommunications, IEEE Journal of 
Selected Areas in Communications, Vol. 9, No.4, May 1991, for low values of 
delay (that is, for D smaller than 250 ms), and over-estimates the MOS 
degradation for values of Z) that are larger than 250 ms (that is, for round-trip 
times that are larger than 500 ms). Hence, one embodiment that corresponds to 
this embodiment uses the following metric: 

exp(- 1 . 1 D) £ mvD ^ exp(-2 .OD) 

with an average of 


mvD = exp(-1.5D) 

That is, a., = 1 .5, p.. = 23, yielding 5„ = 15. Also, using the bounds for a., and p.. 
obtained above, the corresponding minimum and maximum values for 6„ 
become: 10<6,, <23. 

Metric for video traffic 

In this section, we describe the derivation of one embodiment of a 
metric for video. In this embodin^ent, we use a model that is very similar to lhat 
used for voice. Indeed, it is assumed in this embodiment that the degradation of 
a voice conversation because of excessive delay is very similar to the 
degradation of a video communication. In applications such as video- 
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conferencing, the aural sense (the fact that one end is listening to the other end) 
is complemented by the visual sense (that is, the fact that one end also sees the 
other end). As a result, in some embodiments, the video metric can be 
considered slightly less sensitive to delay than the voice metric. 

As far as loss is concerned, in some embodiments, the voice metric 
underestimates the effect of loss in video quality; indeed, in such embodiments, 
the loss of one video frame can affect a large number of frames. The actual 
number of frames affected depends on the encoding of the video sequence. In 
this embodiment, we use the concept of useful frames to derive, from the metric 
obtained for voice, a metric that can be applied for video. A useful fi*anie 
denotes a frame that is successfully decoded at the receiver. In some 
embodiments, the effect of loss on the metric can be increased by taking into 
account the average number of frames lost by the encoder upon the loss of a 
frame as video traffic traverses a path (that is, in some embodiments, as video 
traverses the network). In such an embodiment, the ratio of frame loss (that is, 
those frames that the receiver is unable to decode successfully) vs. packet loss is 
used to obtain a scale factor that can be applied to the loss component of the 
voice metric function, yielding the loss component of the video metric. In this 
embodiment, the following methodology is used to derive this ratio. First, a 
model for frame loss along the path is assumed. In this embodiment, it is 
assumed that all frames are affected by lost independently, e.g., the loss model 
is Bernoulli. Those skilled in the art can follow the same methodology and 
derive a similar metric for video for other loss models considered. (Specifically, 
in some embodiments of this invention, it is useful to consider a loss model that 
is clustered, following a model of loss that is attributed to the Internet.) The 
independent model can, in some embodiments, be applied to the Internet, since 
every frame is typically formed by more than one packet; that is, in such an 
embodiment, it is assumed that a cluster of packet loss occurs in a single frame, 
hence this independent assumption holds in the Internet. In some embodiment, 
it is assumed that a Group of Pictures (GOP) contains A-frames, one 1 frame, 
and /V-1 P frames. If an I frame is lost, the rest of the A'' -1 frames of the GOP 
are lost; if the first P frame is lost, the rest A/ - 2 frames of the GOP arc affected. 
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and so on. Therefore, in some embodiments, the scaling factor can be computed 


as: 


'S'/^vidco = (l/AO X [1 + 2 + ... + I)] = (A'- l)/2 
Hence, in some embodiments, the video model becomes: 

inelricyidco = exp(-Z) - 69/) 

That is, a„ = 1 .0, p, = 69, yielding 5„ = 69. Note that in this embodiment, the 
value of a,, is smaller for video than for voice, to take into account the fact that 
video is less sensitive to delay. Also, in this embodiment, the value of (3„ is 
scaled by a factor SF,;^,^ = 3, which coiresponds to N = 7. Those skilled in the 
art can derive metrics that use different values of a„ = 1 .0, p„ = 69, depending 
on the set of assumptions used. Also, those skilled in the art can use the 
methodology described above to derive various metrics, based on different 
parameters, depending on the assumptions. 


Metric for TCP data traffic 

In this section, we describe an embodiment of this invention that applies 
for generic TCP traffic. We find the appropriate values for a,, and hence 6, 
in this context. In some embodiments of this invention, the metric is applied to 
short TCP connections. In other embodiments, the metric is adapted to the 
throughput of a typical file transfer, assumed to be 75 KBytes; in yet other 
embodiments, the metric is to be adapted to the throughput of an infinite file 
transfer. Some embodiments that cover two cases: (1) the latency of short 
transactions (e.g., a buy button on some web site) and (2) the throughput of files 
that have a "typical" size of 75KBytes. Those skilled in the art can use a similar 
methodology to derive the metric for files of other sizes for other embodiments. 

Optimizing 5„ to capture the increase in latency for a short connection 

In some embodiments, the performance metric used for short 
connections is the latency incurred from the instant the user asks for the 
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transaction to be sent, to the time the packet containing the transaction arrives at 
the destination. Hence, lo = Lalency{Do)/Lo(ency(P) and // = 
Lalency(0)/Laiency{r) are the corresponding normalized metrics, where / 
represents the one-way packet loss rate on the path, D represents the one-way 
5 delay, and Do is the minimum assumed one-way delay RTT^Jl = 25 ms. In one 
embodiment, we approximate the latter two measures with negative 
exponentials, yielding to mdo and Wdi. respectively, i.e., 

mdD " exp(-adD) Latency(Do)/Latency(D) 
1 0 nidi = exp(-Pd/) ^ Latency(0)/Latency(/) 


Note that the equation for m^o seems to lead to the following discrepancy: 
nijD{Do) '^l, while loiDo) = 1. It appears that this discrepancy could have been 
avoided by using a normalized version of nido. nijoiD) = txpl-adD-Do)] so that 
1 5 nidoiDo) = 1 - However, by doing so, we lose the similarity between the 

performance metrics for voice and data. Also, as will be seen in the following 
graphs, niciD = exp(-a^ D) approximates Id{Dq) quite well. Hence, in this 
embodiment, there is no incentive in normalizing nido in this fashion. In other 
embodiments, normalizing mjo could be warranted. 
20 We start with an approximation of /o, that involves the derivation of a single 
constant a,/. This is done in Figure 6a. As can be seen from the figure, /;7,/d(25 
ms) 1, as expected. Approximating Id with mjo leads to an overestimation of 
the performance degradation for low one-way delays (that is, for D < 100 ms) 
and very large delays (that is, for D > 400 ms). However, the approximation is 
25 quite reasonable for D in the [0, 500 ms] range, that is, for RTTs ranging from 0 
to one second. The corresponding value of is 4.5, yielding to m^^o = exp(- 
4.5D). Using the value a^/^ 4.5 thus obtained, we now approximate // using 
ni,/o nh/f = exp(-a,/A) - PdO; the result is shown in Figure 6b. the approximation 
seems reasonable in this case too, leading only to a slight underestimation for 
30 very low (that is, lower than 1%) and very large (that is, larger than 8%) loss 
rates. The resulting value of p,/ is 16, and the corresponding m,n becomes m,n^ 
exp(-16/). 
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•n,«afore. in some embodiments, tte corresponding value of 8. in this 
case is 16/4.5-3.6. Thar is. for such embodiments, the value of 6 is much 
■ower for data traffic than for voice traffic (in which case 6 was. as shown above 
equal to 1 5). In fact, in one embodiment with TCP tmffic. the relative 
unportance of delay as compa,.d to loss is much higher. ,„ one embodiment 
w..h v,deo traffic, ,% loss rate was epuivalen, to a ,50 n,s delay, whereas in 
one embodim«« with TCP trafBc, a ,•/. ,oss.te is only worth a 36 ms delay 
Conversely, a 1 00 ms delay in one embodrment with voice t.ffic is e,uiva,em 
o a mere 0.6% loss rate, whereas a , 00 „,s delay in one embodiment with TCP 
tratfic IS equivalent to a 2.78% loss rate. 

™ approximations for a rnnge of loss rates and one- 

- delays ,„ order ,o unde.,and the effectiveness of our appro.imat.ons when 
Mb pace, loss and delay come into play concurrently. Our approximation 
does surpnsingly well in n.ost of the ™ges of interest for delav and loss rate 
respectively. ■ 

0„g 5,„„ eapt„„ «,e decease i„ ,b.„„ghp„, e„„„ec.,.„ of 

typical file size 

Where tl^e ' '° '° '"'^ »™ ™'^o*-".s. 

Where he metnc negattve exponential n,etric captures the decrease m 

throughput for a connection of typical me si^e. ,„ such embod.ments the 

descnbeda ove./,eprese„,stheo„e.waypaclcetloss.,eo„ thepa.b Z> 

represents the one-vvav delnv anH n • .1 . . ' 
Rm. - . " " "'"'"""^ '-"^^"'^^d one-way delay 

^^/o/2-23ms.AssLimincthatvvearenhIPfoo ■ 

^ ai e able to approximate the latter two 

meastires with negative e.xponentials vieldin. to . h 

must hence have ' ''^^P-'-^'y. -e 


nMD = e.xp(-adD) . Tlirot,ghp.t(D)/ Throughput (Do) 
"^dl = exp(-PdO - Throtighput (/)/ Throughput (0) 
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Note that, as in the previous sub-section, the equation for mjo seems to lead to 
the following discrepancy: mjo{Do) ^^l, while lo{Do) = 1. It appears that this 
discrepancy could have been avoided by using a normalized version of ;;/,/d, 
5 mdoiD) = exp[-aj(D-Do)] so that /7/,/o(Z)o) = 1 . However, for the same reasons 
described in one embodiment with short TCP connections, some embodiments 
(described here) find no incentive in normalizing Wdo in this fashion. (Other 
embodiments could, on the other hand, find it useful to normalize mjp in this 
fashion.). 

10 In some embodiments, an approximation of /o can be found, which 

involves the derivation of a single constant a^/. (See Fig. 8a.) As for short TCP 
connections, ;;vo(25 ms) ^ 1, as expected. Also, the approximation is quite 
reasonable for D in the [0, 500 ms] range, that is, for RTTs ranging from 0 to 
one second. The corresponding value of a^f is 6.5. yielding to nido ~ exp(-6.5i)). 

15 Using the value of a^^ 6.5 thus obtained, // can be approximated using mdo-fn^n 
= exp(-ac/Do - PdO; result is shown in Figure 8b. The approximation seems 
reasonable in this case too. The resulting value of p^/ is 37, and the 
corresponding niji becomes lUd! = exp(-37/). 

Therefore, for such embodiments, the corresponding value of 5d is 

20 37/6.5 = 5.7. That is, when these assumpfions hold, the value of 5 is 

significantly higher than for short TCP connections (3.6), but still lower lower 
than for voice traffic (10 minimum, 15 on average). In fact, in one embodiment 
with TCP traffic, the relative importance of delay as compared to loss decreases 
as the file transferred increases. In one embodiment with typical file transfers, 

25 1 % loss rate is equivalent to a 57 nis delay, which is in between the 36 ms delay 
obtained in one embodiment with short TCP connections and the 150 ms delay 
obtained in one embodiment with voice traffic. Conversely, a 100 ms delay in 
one embodiment with typical TCP connections represents a 1.75% loss rate, in 
between the low 0.6% loss rale obtained in one embodiment with voice and the 

30 large 2.78% loss rate obtained in one embodiment with short TCP connections. 
In order to understand the effectiveness of these approximations in such 
embodiments when both packet loss and delay come into play concurrently, we 

17 


wo 02/J3896 

PCT/USOl/32476 

Show in Figure 9 the approximations for a range of loss rates and one-way 
delays. As for short TCP connections, our approximation does surprisinmy well 
m most of the ranges of interest for delay and loss rate., respectively. 

Unified metric for TCP traffic 

In summary, the graphs above show that, in some embodiments the 
appropriate values of 6. for short connections and typical file transfers are 3 6 
and 5.7, respectively. Jn some embodiments, it is desirable that one value of 5, 
be applied to a number of TCP applications; in some embodiments. 5. can be 
set to the average of both values, 4.6. In other embodiments, it could be 
desirable to use different values for each. Using a similar methodology those 
skilled in the art can derive embodiments with different parameters, and 
different values for these parameters. 


Unified metric for both voice and TCP data traffic 


Summarizing our results 

Table 1 below summarizes the results described in the previous two 
sections for some embodiments. Other embodiments for voice traffic can take 
into account the effect of clip length. 

Table 1 Summarj' of values for this embodiment 


1 raffle Type 

a 

'T 

'1 

Voice 
ICP 

!.:> 


ID 





Short TCP 
connections 


10 

3.6 

typical TCP 
1 transfers 

6.5 

_. 

J/ 

5.7 


In some embodiments, it is desirable to use one metric for both voice 
and tcp traffic; in at least one i 


above with 5=10 can be used 


or more of such embodiments, the metric describe 
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Deriving user-perceived performance measures for web applications 

In some embodiments, the metric measures the quality of specific 
applications used by humans, that use TCP for transport. For some of these 
embodiments, adequate performance metrics measure the model the subjective 
user-perceived quality of the application. Hence, in some embodiments, TCP 
performance can be mapped to objective application performance, which, in 
turn, can be mapped to user-perceived quality metrics. In some embodiments, 
the application of interest includes a web transaction (http). Other embodiments 
can focus on constructing a metric for other such applications, such as telnet, 
ftp, or other. In some embodiment, a mapping between the web transaction 
duration and the underlying latency of a TCP transaction can be derived and 
used. Using this model, the duration of a web transaction can be found as a 
function of the network performance metrics (e.g., delay Jitter, and loss). In 
turn, for some embodiments, the duration of a web transaction can then be 
mapped to an application score, which can take into account the subjective 
perception of the application quality by the user. The sequence of such 
mappings is shown in Figure 10. In this document, we provide the insights 
behind some embodiments, that involves the choice of models at each of the 
steps shown in the procedure. We start by describing the model used for TCP 
transactions that corresponds to some embodiments. We then go over the 
specifics of the HTTP model used (yielding the duration of a web transaction) 
in some embodiments. Finally, we explain how, for some embodiments, the 
duration of a web transaction is mapped to a user-perceived metric. 

TCP models 

TCP functions and mechanisms 

In the following, we describe the Transport Control Protocol (TCP) 
modeled in some embodiments. TCP is a window-based protocol wdiich 
principal function is to provide a reliable transport between two end-points. It 
starts with protocol handshake, followed by the transmission of packets using 
a sliding-window algorithm, according to which the sending window advances 
as acknowledgments are received. A simple TCP transfer is shown in Fig. 1 1 . 
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TCP is also provided with mechanisms that render its utilization of resources 
(that is, bandwidth and buffer) in the network adaptive, depending on the 
conditions in the network (e.g., loading, congestion, etc.). These mechanisms 
are slowslarl, congestion avoidance, fas I retransmit, and fast recovery. In the 
following, we briefly describe each of these phases. More details can be found 
both in the original congestion avoidance paper by Jacobson V. Jacobson, 
Congestion Avoidance and Control, SIGCOMM' 88.and in the invaluable book 
by Stevens W. R. Stevens, TCP/IP Illustrated, Volume 1: The Protocols, 
Addison- Wesley, 1 996. 

Protocol handshake 

A TCP connection is preceded by a three-way handshal<e: the sender first 
transmits a SW, to which the receiver responds with a STOACK. The sender 
finally replies with an ACK upon the receipt of the SYN/ACK. (See Figure II.) | 

Slow start 

It should be clear that the bandwidth used by a TCP comiection increases with 
the window size used by the sender. Since it may not be known originally how 
much bandwidth is available for the transfer, a start is selecting a small window 
size (typically equal to one segment). After that, it increases the window size b)- 
one for each acknowledgment it receives, until one of the following events 
occurs: 

1 . Either a maximum window size is reached, which is the minimum 
among the default maximum window size used by TCP, typically 64 
KB, and the receiver socket buffer size (which can be set by the 
application, and which default varies depending on the operating 
system). 

2. Or the rate of packets sent is larger than the available bandwidth, leading 
to a packet loss. In this case, the window is set back to ] . A packet loss 
is detected using one of two ways: either the TCP retransmit timer 
expires, or three duplicate acknowledgments for the same sequence 
number of received. More details about the detection of loss can be 
found in the fast retransmit and fast recovery section. 
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In both cases, a new phase is entered, one in which a combination of slow start 
and congestion avoidance are performed, as described below. 

Congesiion avoidance 

After either the window size reaches its maximum, or the loss of a packet 
occurs, the TCP transfer enters the congestion avoidance phase. In this 
embodiment, TCP is assumed to transmit at a given window size cmn1= F(the 
sender window size is called the congestion window size, cwnd), and that a loss 
event occurs. At this point, halfiht value of the current v^'indow W is saved into 
a variable called the slow start threshold (ssthresh = WI2\ and c\md is reset to 
1 . Thereafter, TCP enters the slow start phase, until cwnd reaches ssthresh, at 
which point TCP enters the congestion avoidance phase. During this phase, and 
as long as no packet loss is detected, cwnd is only increased by Mcwnd at each 
ACK received, leading to an approximately linear increase in the window size 
with time. 

The rationale behind this behavior is as follows: the original window size was 
clearly too large, since it resulted in a packet loss. Hence, a lower WMudow size 
must be used, and this window size is most probably somewhere between Wl2 
and W. Accordingly, slow-start allows c\md to reach W/2 vei7 quickly (i.e., 
exponentially); at this point, a bandwidth discovery process is initiated, in 
which TCP attempts, in a greedy manner, to obtain as much bandwidth as it can. 
In case a packet loss occurs again at any step of the process, cmid is reset to 1 
again, and the process is restarted again (i.e., slow start followed by congestion 
avoidance). 

Fast retransmit and fast recovery 

In our discussion of congestion avoidance, we did not differentiate between 
types of packet loss. However, the TCP sender detects packet loss using one of 
two methods: 

1 . Timeout: Once TCP sends a packet, it starts a timer RTO. If the ACK for 
that packet is not received at the expiry of 7270, the packet is recorded 
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as lost. RTO is computed in one the following ways for some 
embodiments: 

• During handshake, the time-out is initialized to a large value for 
example either 3 seconds in some embodiments, 5.8 in some 
embodiments. 

• Once Round-trip time measurements are obtained, then RTO is 
obtained using the following formula in some embodinients: 

RTO = A + 'iD, 

Where^ is a moving average of the round-trip time, while D is a 
mo.ng average of the mean deviation of the round trip time. i.e. the quantity 
-Al where M denotes the latest round-trip time measurement obtained at the 
sender side. More specifically, for each packet received, A and D are obtained 
as follows (even though A and D are typically only updated once every 500 ms). 

A^{\-g)A+gM 

Where g is a small gain factor (which, in some embodiments is set to a 
negative power of two). 

2. Triple duplicate ACK.: i, case a packet is lost, the receiver wii, generate 
for each following segment received an ACK for the sante sequence 
number, which ccresponds to that of tl,e lost packet. Hence, in case the 
sender receivers a multitude of ACKs for the same sequence nu,nber it 
can assume that the corresponding packet was lost. In TCP three 
duplicate ACKs (that is, four ACKs for the same sequence '„„„*e,) 
s.gnal a packet loss. Only in this case do the Fast Retransmit and Fast 
Recovery algorithms kick in for some embodimeras 

Heuce. once a triple duplicate ACK is detected, the sender ente. .wo processes- 

.t first retransmits the segment it believes is lost. This is called a 

lie,n,m,m (since TCP doesn't wait for a time-out to occur before 

retransmitting). Then, as in congesiion avoidance, „ se, to half the 

current window Howpvr^r rr^n « x 

lou ever, TCP sets cwncl to s,llvc,h in this case, hence 

avoidmg the slow-start phase. 
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Typical (and simplified) traces of the TCP window size in time are shown in 
Fig. 12 for two cases: in one case, transmitting at the maximum window seldom 
saturates the link. This can be the high bandwidth case; effective bandwidth can 
be translated to a scale that is consistent to that of window. That is, we can 

5 actually be showing the product of effective bandwidth and round-trip time. 
Hound-trip time can assumed to be constant in the figure (if not, then the 
window size increase may not have been linear with time during the congestion 
avoidance period)). In tlie other case, the appropriate ideal window to be used 
is significantly lower than the maximum window size chosen by the source. 

10 Clearly, the difference in TCP behavior between the two cases is vei7 
significant. In particular, the only way the window size is limited in one 
embodiment with the low bandwidth path is through packet loss. Combined 
with TCP's greediness in its attempt to capture more bandwidth from the link, 
this fact leads to a systematic pattern of packet loss. 

15 TCP performance 

The description of TCP's mechanisms above is helpful in understanding 
how a performance model for TCP is derived. Here, the model for TCP latency 
derived in N. CardwelL S. Savage, and T. Anderson, Moclding TCP Laiency. 
IEEE INFOCOM 2000, April 2000. is reviewed, as we believe it to be the 
20 appropriate in capturing some embodiments owing to its accurate modehng of 
the protocol handshake and slow-start. The equation below summarizes the 
findings of N. Cardwell, S. Savage, and T. Anderson, Modeling TCP Latency, 
IEEE INFOCOM 2000, April 2000. 
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(Slow -Start) 


^7T[|r(;,)+ij+e(p,,,,(p))G(£)£ro 


. ^ J'^ max / \ ^ p 


l~ p W ~ 
~ + ~^ + 0{p,W ) 


(Transfer of remaining bits) 
+ (Delayed Acks) 

where 

^0 = Tj = 3 seconds 

RTO = RTT + 4 X mdev(RTT) 
y=l+(]/b} 

Efdss] = ((l-(I-p)'')(i.p)3/p 

Q(P.v) = .inO,0.(,.p,3,_^^.^^.-3^^^^^^_^^_^^^,^^^^ 

G(p)=]+p + 2p- + 4p^8p%16p^32p^ 

W(P) = ((2 . b)/(3b)) . Sq.«(8(, . p,y,3 b p)) . ((2 . b)/(3br) 

Daxk= 100 ms 

AS Shown in ,he equation above, ,„e latency of a TCP connection L 

depe„dso„t„efi,esi.etobetransfe„.ed.(innun,ber„fsegn,en,s, t " , , 
iound-lriptinieK7Ttl,eDact,l »Tr -,. , ==6'™"'=). the paol;et 

„ Zi, ,■ . '^ ""^•""^ ='"<ltl>epaclcet loss rate 

L .3 co„s.,.uted of vaHons components, wh.ch are. in order, the latency 
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associated with the handshake, the slow-start, the first loss, and the transfer of 
the remaining bits. We describe the specifics of each and then include 
comments pertaining to the overall equation. 
Protocol handshake 

5 This equation states that the average time needed to complete the protocol 
handshake is one round trip time, to which a timeout is added to each lost 
packet in the transaction. Note that at this stage, the timeout is typically as large 
as 3 seconds for some embodiements, so even a small loss probabilit)' can be 
significant. 

10 

Slow slaii 

As described above, slow start lasts from the end of the handshake up to the 
time the window size ceases to increase (whether because a packet was lost, or 
because the window has reached its maximum value). In the context of N. 

1 5 Cardwell, S. Savage, and T. Anderson, Modeling TCP Latency, IEEE 

INFOCOM 2000, April 2000, in an attempt to make the math simpler, slow start 
is given a slightly different meaning. In fact, slow start is defined as the time 
period separating the end of the handshake up to the first loss, independently of 
whether the maximum window size flw is reached. Given the packet loss p 

20 and the dynamics of the window size increase during slow start, both the 

expected number of segments sent during slow start E{dss] and the window size 
reached at the end of the slow start process E[W,s] are computed. Clearly, as 
shown in the equation, in case E[Wss\>Wn^ax. then the latency calculation is 
divided into two periods, the first capturing the exponential window growth, 

25 and the second capturing the period in which the window is constant, equal to 
Wtnax, It is interesting to note that in case packet loss is null, then slow start 
according to this definition lasts for the entire length of the connection. 
Conversely, in case E[I-K,.J<rF,„,,v. then the latency associated with slow stait 
only includes a single component that corresponds to the exponential increase in 
30 window size. 
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First loss 

As described above, a packet loss is detected vvitliin some time period 
that translates directly into additional delay incurred by the user. This delay 
depends on whether the packet was detected through a time-out or a triple ' 
duplicate. In case of a time-out, then the delay is equal to RTO, which in this 
case is set to the initial 3 seconds in some embodiments. In case of a triple 
duplicate ACK, the extra delay is as low as one round-trip time. In the equation 
above, the probability that packet loss occurs because of a timeout 0(p, is 
computed, given the loss ratep and the instantaneous window size w. The 
latency associated with the first loss can hence simplv be computed as a 
weighted sum of RTO = and RTT. Since a loss event can result in more than 
one packet loss, successive time-outs can potentially occur; the factor G(p)/0 - 
P) takes into account this fact by appropriately scaling the first time-out value 
To. 


Transfer of remaining bits 

The following component of the TCP latency equation corresponds to 
the latency incurred by the transfer of the bits that remain after this first loss 
Although complex, we can see that the formula basicallv models the effect of 
loss (whether detected through timeout or the receipt of triple duplicate ACKs) 
and that of congestion avoidance (as witnessed by the computation of Wip) 
wh,ch represents the average window size given the loss rate and the dvnamics 
of congestion avoidance). Hence, this component of the equation is basically the 
ratzo of the remaining bits to be transferred (^- ^^j) to the average rate of 
transfer, in turn equal to one window per time period set to the sum of T^TT and 
whatever additional delay is caused by congestion avoidance. What this 
component does NOT^model, however, is slow sta. after retransmission of 
tuneouts. This means that the equation above assumes that TCP is more 
aggressive than it actually is. which results to an under-estimation of the actual 
latency involved in the transaction. However, N. Cardwell, S. Savage and T 
Anderson, Modeling TCP Latency, IEEE IKFOCOM 2000. April 2000, e.xplains 
that the effect of this negligence should be small. 
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Delayed ACKs 

The final component in the equation above models (in a trivial way) the 
extra delay caused by delayed ACKs for some embodiments. In fact, 100 ms 
5 was measured as a good average delay increase caused by delayed ACKs in 

some embodiments. ACKs are not always sent back to the sender immediately 
after a segment is received at the destination. In fact, the functionality of TCP is 
as follows for some embodiments V. Paxson, Ainomated Packet Trace Analysis 
of TCP impleineniations, SIGCOMM' 97: two packets have to be replied to 

1 0 immediately. However, the receipt of an individual packet is only replied to at 
the end of a clocking timer, typically set to 200 ms in some embodiments. 
Hence, 100 ms seems like a good average value to be added to take into account 
the effect of duplicate ACKs for some embodiments, 
(e.g., see M. Allman, A Web Server s View of the Transport Layer, ACM 

1 5 Computer Communication Review, Vol. 30, No. 5, October 2000. 

Comments about the model 

The model in i\4. Mathis, J. Semke, and J. Mahdavi, The Macroscopic 
Behavior of the TCP Congestion Avoidance Algorithm, ACM Computer 
20 Coinniunication Review^ July 1997, is indeed simpler, but also captures much 
less of the richness inherent to the behavior of TCP. As shown above, the 
model captures, in an intuitive way, a lot about TCP's behavior. In this section, 
we go over straightforward and useful comments on the model. 

] . Latency does not depend on the link bandwadth. In fact, this model does 
25 not directly model the link bandwidth, but considers it through its effect 

on delay, jitter and loss. For example, if bandwidth is too low, then the 
loss rate increases (as is clear from Fig. 1 1). At the same time, round-trip 
time increases through increased queue sizes at the bottleneck link TCP 
time-outs. Note that for transactions that involve small file sizes, the 
30 latency is mostly affected by the first two components, i.e." the protocol 

handshake and slow start. Hence, the throughput of the transaction is 
probably unaffected by the bandwidth of the bottleneck link (that is, 
other than the effect of the later on loss rate or 7^77^. 
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2. Latency is in all cases a linear function of the round-trip time. It is true 
that this is an artifact of the above assumption (since the only non- 
linearity would have come fi-om the effect of bandwidth on the latency 
of the TCP connection). However, since the effect of bandwidth alone is 
veo' small, the linearity of the TCP latency with the round-trip time 
holds in most cases in real systems. 
3. Contraty to round-trip time, latency is a non-linear function of loss 
probability. In fact, for loss rates above 5%, the increase in latency with 
the Joss rate accelerates significantly. 

In some embodiments, the model described above can be used to model 
TCP behavior in one or more of the steps. 

HTTP 

In some embodiments, the metric measures web transactions In this 
section, we describe http mechanisms used in some embodiments. In some 
embodiments, such transactions use the hypertext transfer protocol (HTTP) 
wh,ch, in some embodiments defines how TCP is used for the transfer of the 
different components of a web page (that is, for example, the html page the 
different miages. etc). In some embodiments, the first version of HTTP 
HTTP/1 .0, described in RFC 1 945 T. Be,.ers-Lee, R. Fielding, and H. Frvstyk 

1945, May 1996, encourages the following practices: 

1. Different components of a given web page (e.g., the htm. text, and each 

of the different objects) are transferred using distinct TCP connections 
- In order to increase the speed of the transaction, web browsers are 
allowed to open more than one TCP connection at a t.me. (The tvpical 
number of parallel concurrent TCP connections is 4 in some 
embodiments.) 

m 30„,e en,bod™e„,s, ,his approach is adequate for relatively smaller 
- pages, wl,ie„ include a few objects, when traffic on paths „„ the Internet is 
l^ited. But as the ,„te„,et has heconte .ore and .ore „hi,nitous. the 
d.sadva„,a,e3 of this approach can become, in some emboditnents more and 

more apparent: 
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1 . On one hand, in some embodiments, an independent TCP connection 
goes through both the handshake and slow-start mechanisms. Since, in 
some embodiments, individual objects are relatively small, so TCP 
connections spend most of their time in these stages, rather than in tlie 

5 congestion avoidance stage, hi some embodiments, this can be 

inefficient. 

2. On the other hand, it has been shouoi that in some embodiments, 
opening a multitude of TCP connections simultaneously increases the 
greediness and aggressiveness of the web browser's behavior, H. F. 

10 Nielsen, J. Gettys, A. Baird-Smith, E. Prud'ommeaux, H.W. Lie, and C. 

Lilley, Network Performance Effects of HTTP/ LI, CSSl, and PNG, 
SIGCOMM' 97, H. Balakrislinan, V. Padmanabhan, S. Seshan, M. 
Steem, and R. Katz, TCP Behavior of a Busy Internet Server: Analysis 
and Iwprovments, IEEE hifocom 1998, S. Floyd and K. Fall, Promoting 

1 5 the Use of End-to-End Congestion Control in the Internet, IEEE/ACM 

-Transactions on Networking, 7(6), August 1999. To understand this, we 
provide the following simple example (from S. Floyd and K. Fall, 
Promoting the Use ofEnd-to-End Congestion Control in the Internet, 
, IEEE/ACM Transactions on Networking, 7(6), August 1999). Say a data 

20 transfer is divided among parallel, concurrent TCP transactions. 

Assume a packet is lost in one of the connections. If all the data were to 
be transferred using one single TCP connection, the lost packet would 
lead to the halving of the window size, i.e. to the halving of the 
connection tliroughput. Instead, when N concurrent TCP connections are 

25 used, the lost packet will only halve the window size of one of the 

connections, leading to a reduction of the aggregate throughput by a 
mere 1/2A^! That is, the congestion algorithm that TCP is intended to 
perform is skewed towards a much larger greediness and aggressiveness, 
leading to an increase in congestion that can in turn bear a significant 

30 degradation in the performance of all streams involved. 

In some embodiments, the first problem can be solved through the selling of the 
Keep-Alive option T. Berners-Lee, R. Fielding, and H. Frystyk, Hypertext 
Transfer Protocol - HTTP/LO. IETF Request for Comment RFC 1945, May 
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1 996. In some embodiments, an open TCP connection is not closed afier tl,e 
object transfer is completed, but rather used for tIre transfer of the next object 
Th,s way, in some embodiments, the number of concurrent TCP connections 
_ remains equal to four. In some embodiments, the latency of the total „eb 
transfer can be improved by reducing the number of mandatory protocol 
handshakes and slow-sta*. However, in some embodiments, the Keep.,,... 
optton does not solve the second problem. ,„ fee. „ such embodiments, settin. 
h ^ee,.A,n.e opt.on only exacerbates the second problem, and renders .he ~ 
behavior of the web browser even more agg,essive. 

lnlightofsuchf;ndi„gs,arevisi„„ofHTTO(HrrP/,.l,R.Fie,di„„ J 
Oettys. y Mogul. H. Ft^styk. and T. Ber„e..Lee, H,,er,e. Transfir ' 
Pro o , IETF Request for Comntent RFC 206S. Janua^ ,997) 

Eluded new ways to be used by the web-browser for the transfer of a web 
page ustng TCP. HTTP/, . 1 advocates the use of a single TCP com^eCionfor 

.etransferoftheentirewebpagcsuchacomtectionis called persistent 
Thrs way. the second problem described above is solved and TCP recover 

..s ortgina, aggressiveness. ,„ addition, the clientcan pipeline requests for 
different objects belonging to the same web page o„ the same TCP 
connection. This way. the transfer of th« 

Which has the desim le etT ct " T"" ''''''' ' 

„ . . , of reducing the variability between the delav, 

associated to the download of each of the different objects 

Assimre a web page includes the combination of text and 9 images. In Fi,. 

-H.edi.eren.,.ystotransferawebpageinsomeembodLe„. : 

Insome embodiments, assuming i„fi„i,. 

HTTP/1 .0, with the Kee^,n,e option set basicallv divides the time 1 1 

download the images by a factor of 4 I r„ ' '° 

-ming the .alistic c se wh^^^ . 

web transaction can be assume J r"'" 

individual TCP connections; that s t or T: ^"'""^ 

^^'C. ^A/////, followed bv the rmvi'm., ™' 
' "'^^"^"'^ among the 4 TCP connections, A„„ = 
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max(Li, Li, L3, La). In some embodiments, finding the exact maximum among 
various TCP connections is a complex process, as it implies knowledge of the 
latency distribution, whereas the equation described above only derives the 
average latency. In some embodiments, Lniax<^^ be approximated, based on the 
5 foUov/ing assumption: assuming that only one coiuiection among the four 

connections incurs a packet loss, then L,nax may simply approximated to be the 
duration of that particular transaction. Assuming that the loss probability on the 
link is equal to p, each connection then may be assumed to incur a loss rate p, so 
that the probability of no loss by either connection can be approximated to = 

10 1- P = i^'pf- That is, in some embodiments, the probability that either 

connection experiences a loss becomes equal to P = 1-(1 which, in some 
embodiments, can be approximated to 4p for small p. That is, in some 
embodiments, an approximate value of Z,;;,av can be obtained using the equation 
for average latency, and in some embodiments, the loss rate can be set to 4/?. 

1 5 Other embodiments may use different approaches to yield various models. 

Description of models used 

In the foUov/ing, we describe different embodiments of models that can be 
derived for both HTTP/1.0 and HTTP/1.1 downloads. In some embodiments, it 
can be assumed that what is downloaded is one or more plurality of objects 
20 from what is assumed to be a "typical" web site. Other embodiments of this 

invention can derive these results for other HTTP embodiments , and other web 
sites. 

In some embodiments, a typical web transaction can be modeled using the 
following: 

25 1 . A one-segment request for the web page to be transfer. 

2. ... followed by a transfer of a file of size / In some 

embodiments, the actual latency formula representing the 
transfer depends on whether HTTP/1.0 or HTTP/1.1 are used. 
In some embodiments, we can assume that the path is characterized by a loss 
30 rate p, a round-trip time RTT, and a round-trip lime jitter J. 

In the case of HTTP/1.0, some embodiments assume that the Keep-Alive option 
is set. Also, some embodiments assume that half the file constitutes the actual 
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html ASCII text, whereas the other half includes the different images in the file. 

While, in some embodiments, the ASCII is downloaded using a single TCP 
connection, the information pertaining to the images is downloaded, in parallel 
over 4 independent TCP connections. Denoting the latency of a TCP transfer of 
a file of size/., over a path characterized by a loss rate p, a round-trip time RTT 
and a round-trip time jitter Jby L{f,,p, RTT, J), then the total duration of the 
web transaction can, in some embodiments, be approximated to 

A^TTP/,.0 = ^C1500Bytes,/7, RTT, J) + L(f,/2,p, RTT, J) + L^, 4p, RTT, J) 

Those skilled in the art can start with a different set of assumptions, follow a 
similar methodology, and derive various other models that derive the total 
duration of an HTTP/I .0 web transaction in function of the various dynamic 
parameters of the path. In the case of HlTP/l.l, some embodiments assume the 
use of persistent, pipelined transactions. Since, in some embodiments, a single 
TCP transaction is used for the download of the entire file, the total duration of 
the web transaction can, in some embodiments, be approximated to 

Amp/,.i = I(1500Bytes, A RTT, J) + L(J,,p, RTT, J). 

Those skilled in the art can start with a different set of assumptions, follow a 
snmlar methodology, and derive various other models that derive the total 
duration of an HTTP/,., web transaction in function of the various dynamic 
parameters of the path. 


Some web transacdon duration results 

In this section, „e show a set of web t, a„sac.ion duration (denoted D ) 
results for sonte embodiments of this invention, as a tunCion of both round-trin 
..me, loss rate and jitter. (See F.gs. 15-17.) 1„ the come.x, of these examples ,he 
sraphs are shown as a function of one-way delay andjitter, simply ob.ainedhv 
ha vmg ,he round-trip time and round-trip time J.tter, respectively, ^ose skilled 
the ar, can derive examples, where the parameters in the graphs constitute 
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round-trip parameters. Since, in some embodiments, TCP latency as shown in 
the previous section may be linear with round-trip time, the linearity of web 
transaction duration with one-way delay is expected as a result. (See Figure 15.) 
Also, it can be seen from Figure 1 6 that for some embodiments, the curve 
showing the dependence of D„. on p may be convex, that is, its slope increases 
with increasing values of p. Since, in some embodiments, HTTP/1.0 depends on 
4p, the increase of Ar with p may become increasingly significant as p exceeds 
5%. Conversely, in some embodiments, loss rate does not seem to affect much 
the duration of an HTTP/1.1 web transaction. Finally, Figure 17 reveals that in 
some embodiments, the dependence of on jitter is very small for both 
HTTP/1 .0 and HTTP/I .1 , irrespectively of the file size; therefore, in some 
embodiments, the jitter parameter may be ignored. 

metric versus web transaction duration 

In this section, some embodiments of the remaining task of mapping 
transaction duration to a measure of quality that captures the user's perception 
are described. The embodiments described rely in part on relevant papers on the 
subject, A. Bouch, N. Bhatti, and A. J. Kuchinsky, Quality is in the eye of the 
beholder: Meeting users' requirements for Internet Quality^ of Service. To be 
presented at CHl'2000. The Hague, The Netherlands, April 1-6, 2000, pp. 297- 
304.and A. Bouch, N. Bhatti, and A. J. Kuchinsky, Integrating User-Perceived 
Quality into Web Server Design, HP Labs Teclinical Report HPL-2000-3 
20000121, which present experiments designed to estimate users' perception of 
quality in the context of web transactions. (The paper stresses, in particular on 
e-commerce.) In the papers, the authors have created a web site with latency 
programmability. (That is, they control the latency of the transfer of individual 
pages.) Users are asked to go through the different pages of this site, and rate 
the latency obtained as low, average of high. The final result is the graph shown 
in Fig. 1 8, that shows the percentage of users responding "low", "average" and 
"liigh" versus the actual duration of the web transaction. 

The results shown in Fig. 18 can be translated to some subjective 
measure of user-perceived quality, which we denote by "synthetic Mean 
Opinion Score" (MOS), from which the metric will be derived. In this respect, 
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in some embodimenls, a graph presented in N. Kitawaki and K. Itol, P„re 
Delay Effecs o„ Speech OualUy i„ Taleco,„..unlca„ons, IEEE Journal of 
Selected Areas in Commumoations. V„,. 9, No.4, May 1 991 , can be used .ha. 
niapa MOS values .o an "i.perraissible ra.e", ,ha. is. .„ ,he percentage of use,, 
tha. th,nk the quality is unaecep.able. This g,.ph is shown i„ Figure ^9 For 
some embodiments. A MOS versus «b .ran^cion duration function can be 
ob.a,„ed by assuming .ha, .he "high latency" ra.e in Figure 1 8 can be 
.nterprered as an "impermissible ,a.e". ,„ some emb„dimen.s. .his assumption ,s 
reasonable, since, as far as ,e con.en. provider is concerned, high latency is 
unaccepra,, and should never be experienced by the user. This may be thought 
. be true, especally for high value transactions, such as trading, or even 
shopping. Those skilled in «,e art can fo„ow the me.hodo.ogy described in .his 
document to use other studies of user-perceived quality and derive 
corresponding metrics. 

Tlre resulting MOS versus latency curve can be obtained for some 
embod.me„ts. Interestingly, .he curve shows that for these embodime„.s. the 
i^OS does not degrade smoothly as transaction duration increases, h, fac, in 
* se embod,ments a sharp decease ft.™ a MOS of 5 to a MOS of 4 occurs 
wl,e„ the .™sac.ion duration exceeds the two seconds mark. ,n one 
en,bod,men.. the MOS ratings ftom , to 5 bear the following inte^reraHons- 5 
for Excellent. 4 for Good. 3 for Fair. 2 for Poor and , for Bad. Other 
embod-menls use different values. Similarly. i„ some embodiments a similar 
behavor occurs around the 8 seconds mark. "s.as,m,lar 

In some embodiments, deriving the metric from the MOS value can 
comprise a step where the 1 to 3 Mnq.„i ■ ^"luecaii 
from 0 to I In Fi„s .0 , ' ' 

embodiments of this invention. 
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In some embodiments, such MOS functions can be used to derive 
metrics both for HTTP/1 .0 and HTTP/1 . 1 . For some embodiments, the 
following conclusions can be drawn, as observed from the metric graphs above: 

• For some embodiments, the effect of jitter is negligible. 

5 • In some embodiments, the effect of loss rate is not as large as one would 

expect. In fact, in some embodiments, TCP latency increases 
considerably as the loss rate increases from 2% to 10%. However, in 
some embodiments, what is important is the user perception. For 
example, even though an increase of latency from 1 00ms to 2 seconds 
10 represents a 20-fold increase, this increase can, in some embodiments 

have a negligible effect of quality. 

• In some embodiments, The results also show tliat the file size affects the 
metric significantly, which, in these embodiments, is expected. In some 
embodiments, if the focus is on web transfers of the transactional type, 

1 5 then file sizes of up to 10 KBytes may be considered. 

• In some embodiments, the version of HTTP used has a surprisingly high 
effect on the results. In some embodiments, this is especially the case 
when loss rate is high (that is, larger than 5%). In some embodiments, 
the reason for this behavior is, again, the fact that the effective loss rate 

20 incuired in one embodiment with HTTP/1.0 is four times that incuiTed 

with HTTP/1.1. 

An additive embodiment ofHTTP/LO andHTTP/LJ metrics 

In some embodiments of this invention, it is desirable to derive metrics 
for HTTP/1 .0 and HTTP/1 . 1 that are additive (in the sense described above). In 

25 this section, we describe such embodiments. 

The metrics derived here match for some embodiments, those shown 
graphically in the previous section for a wide range of the elementary dynamic 
parameters of delay, jitter, and loss. Those skilled in the art can use the same 
methodology to derive similar metrics with different parameters. 

30 Let a, b, c, and d be parameters that can be tailored to the particular application. 
The embodiments described here correspond to HTTP/ 1.0 and HTTP/1 .1 , 
respectively: 
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For HTTP/1 .0,a= 1.18, b=0A2,c=0.\ 5, and d =0.25; 
Fpr HTTP/1 . 1 , ^ =1 .30, Z> =0.3 1 , c =0.4 1 , and ^ =1 .00. ' 

Let 

X = -(20.0/9.0)*(6 - c); 
.v = (1.0/9.0)*(10.0*c-Zj); 

Let;7.„,, = .Y X delay +y (where delay is the delay in seconds). 
Let loss be a measure of loss on the path. 

^n/oss /d), then nWricLoss = 1.0 - iog(1.0- loss/d)nogO .0-;.„.,, /^) 
if >;7,„a^ /i/), imtricLoss = 0.0; 

melhcDelay = ] .0 - (rfe/q)/)/fl 

Let «,./;-/c denote the r^etric derived for the application; then for this 
embodiment, the value of ,„e/nc is obtained as follows: 

If (mefricLoss + mosDelav > 1 0"> mph-ir - • r 

-* ^ - 'nelncLoss + metricDelav - 1 0 

^0 If imetricLoss + mosDelay ^1.0), ,„,/nc = 0.0. 

The metric value shown in these embodiments is clearly additive, in the sense 
descnbed above. Those 3.„ed in the art can derive additive me^ 
appheafons using different parameters, or different values of these parameters 
f^'lowmg the methodology described above 

In some embodiments, the .etric can, in addition to being performance related 
as descr,bed above), can include non-performance related characteristics- in 
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Apparatus for Characterizing the Oiialit}> of a Network Path 

In this final section, we describe some embodiments of network devices 
that characterize the quality of network paths and use these in the process of 
routing. Effectively, such a network device configured, such that if the network 
5 device is connected to at least a network path including a first segment and a 
second segment, the network device performs: 
1 . accessing a first metric and a second metric, 

- wherein the first metric and the second metric are at least in part 
quality characterizations of a same plurality of one or more 

1 0 network applications, wherein the quality characterization 

characterizes a quality of the same plurality of one or more 
network applications running at one or more segment end-points 

- wherein the first metric and the second metric are at least partly a 
function of a same plurality of one or more elementary network 

15 parameters, wherein the plurality of one or more network 

parameters include one or more of delay, jitter, loss, currently 
available bandwidth, and intrinsic bandwidth 

- wherein the first metric is at least partly the function of the same 
plurality of elementary network parameters of the first segment, 

20 wherein the one or more segment end points include one or more 

end-points of the first segment 

- wherein the second metric is at least partly the function of the 
same pluralit of elementary netv/ork parameters of the second 
segment, wherein the one or more segment end points include 
one or more end-points of the second segment 

2. adding the first metric and the second metric to generate a third metric, 
wherein 

- the third metric is at least partly the function of the same 
plurality of one or more elementary network parameters of the 

0 network path, wherein the one or more segment end points 

include one or more end-points of the network path 

- the third metric is a quality characterization of the same piurahty 
of one or more applications 


25 
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In some embodiments, tiie network device further performs: 
prior to accessing the first or the second metric, generating at least one of the 
first metric and the second metric 

In some embodiments, the network device further performs: 

prior to accessing the first or the second metric, receiving at least one of the first 

metric and the second metric 

In some embodiments, the network device deals with a pluraiitv of one 
or more network parameters that is dynamic. In other embodiments, the network 
device is such ti.at at least one of the plurality of one or more net^vork 
parameters is static. 

As described in the method sections, the plurality of one or more 
network applications include at least one of UDP and TCP applications UDP 
applications include voice, video, whereas video applications include video 
conferencing. In some embodiments, TCP applications include HTTP whereas 
HTTP applications include HTTP/1 .0 and HTTP/1 . 1 . In some embodiments, 
TCP appiications include ftp and telnet. 

In some embodiments, the network device is such that the pluraiitv of one or 
more network paran.eters include delay, jitter, loss, currently available 
bandwidth and intrinsic bandwidth. 

In some embodiments, the metric can, in addition to being perforn.ance 
related (as described above), can include non-performance related 
characteristics; in some embodiments, the non-perfonuance related can n.clude 
pre-specified route preferences. 

In some embodiments, the network device further comprises: 

- a plurality of one or more inputs adapted to be coupled to the 
network path, 

- a.plurality of one or more outputs coupled to the pluraiitv of one 
or more inputs, wherein, responsive to a plurality of one or more 
packets arriving to the network device through the plurality of 
one or more inputs, the network device selects at least one output 
from the plurality of one or more outputs, wherein the at least 
one output is determined at least partly using at least one of the 
hrst metric, second metric, and third metric. 
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Measiiremeni Packets 

A measurement packet is a packet sent by a sender over an internetwork 
that includes information necessary for the receiver of the packet to compute 
measurements of the performance characteristics of the path the packet lias 

5 traversed over that internetwork. The information includes information for a 
receiver of the measurement packet to compute measurements of performance 
characteristics of at least a portion of the path of the measurement packet; and 
data including one or more of measurement statistics, a generic communication 
chamiel, network information, and control data directing a receiver of the 

10 measurement packet to change one or more configuration parameters of the 
receiver. 

In some embodiments of the invention, the information included in the 
measurement packet to compute measurements includes at least one of a 
timestamp of a sending time of the packet and a number to identify the packet 
1 5 by itself and/ to identify the relative position of the measurement packet in a 
sequence of measurement packets. 

In some embodiments of the invention, the measurement packet is 
implemented using the following data structure: 

20 struct MeasurementHeader { 

+ A generation number. This value represents when the 
+ sender began sending. This value is a standard Unix 
25 * timestamp that seconds since Jan 1, 1970 UTC. 

**/ 

uint32_t mGeneration; 

30 * A sequence number for the packet. This increments each 

* time a packet is sent and rolls over when 1 6 bits is 

* exceeded. 

**/ 

uintl6_t mSequence; 

35 
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* The IP address the 
uint32__t mDstAddr; 
/* * 

* The send times tamp 
/ 

uint64_t mSendTime; 


packet is sent to. 


for this packet. 


The mGe„en>.io„ field is u.ed to detect when a sending process has 
started a new session. This tleld is used by tl,e receiver to determine Utata 
dtscontinnity in the streanfs sequence nuntbers is the result of a sender restatl 
rather than due to large network latencies, duplicate packets or dropped paclce. 

The sequence number ™3.<^enoe ti.ia is inc^ented by one each 
inte a packet is sent. This approach allows the receiver to deduce lost and 
dt.pl.cate packets by identifying missing and duplicate sequence numbe. 

The mSendTinte field contains the ttae at which U,e packet was sent 
i-epresen,edas™icrosecondssinceJam,ao.l.l970UTC Thisfieldis ' 
compared to the ti,ne the packet arrived at the receiver to de,e™,„. .he delav 
between the sender and tlie receiver. 

In some embodiments of the invention, a plurality of one or ,™re 
packets are sen. over a path continuously. ,n some embodiments of the 
mvennon, the continuous stream of packet is denoted as a naeasurement stream 
stream is uniquely identified by the source and des nlr 
addresses. The sender maintains one socke, descriptor for each source 

eJI. '""^""''"•"—"'■•-^-"-'--tJdr 
neJd. On tJie receiver side, the soi/rrp fP 

, me soiiice JP address is returned hv the recvO 

s,.s.mcalla,,d.hedest,na.io„addrcssisre,rievedfr„mthemeasurenJ! 
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Data fncliided in the Measurement Packets 

In measurement packets that contain sufficient space, data will be 
included, including one or more of measurement statistics, a generic 
communication channel, network information, and control data directing a 
5 receiver of the measurement packet to change one or more configuration 
parameters of the receiver. 

Some embodiments of the invention will add a single type of data to 
each packet. Some embodiments of the invention will use a complex data, 
including subpackets. 

1 0 Some embodiments of the invention use subpackets that include a single 

byte subpacket type identifier, followed by a 2-byte length field (including the 
length of the type and length fields) and finally including the data that is to be 
sent. One embodiment will store all values in network byte order. Other byte 
orders will be apparent to those skilled in the art. The following data structure 

1 5 definition describes some embodiments. 

class SubPacket { 

/* 

* The type identifier for this subpacket. 

20 */ 

uint8_t mType; 

/* 

* The length of this subpacket, in network byte order. 

25 */ 

uintl6_t mLength; 

}; 

One embodiment of this invention will include data describing a 
30 momentary snapshot of the measurement statistics for a given path belween a 
sender and a receiver. 

In some embodiments of this invention, this data will include one or 
more of the following information: the source and destination IP addresses that 
define the path, a measurement packet size for which the statistics have been 
35 calculated as well as computed measurement statistics that are at least partly 
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«p<,„siv= ,„ del.y; computed „,easureme„< sm.istics are al least partly 
responsive to jitter and co„,pnted measurement statistics t„at are at .east pa«,v 

responsive to packet loss. ' 

1.. one embodiment of this invention, these statistics will be in units of 
n..c.r„seconds e,pressad as 64-bl, tloating-poin, quantities and ..nsm.tted in a 

Standard network byte order. 

In one embodrment of this invention, the following data structure win 
Store the computed statistics: 


class TunnelstatsSubPacket : publ. 


Lie SubPacket { 

The time that this statist: 


**" uijue Tin;^r- t-i-i-;« -.i.^^. . . 

:ic snapshot was taken ( 


microseconds since 1970) " " '^'^ 

uint64_t mTimestamp; 


* to. 

* */ 

uint3 2_^t mSrcAddr; 


* apply to. 
uint32__t mDstAddr; 


* all packet size 
uintl6_t mPktSize; 


wo 02/33896 


PCT/USOl/32476 


* The average delay in microseconds . 
**/ 

5 double mDelay; 

/** 

* The average jitter in microseconds. 
10 double mJitter; 

/** 

* The percentage of packets that have been lost, in the 
range 

15 * 0 to 1. 

**/ 

double mLoss ; 


20 Some embodiments of this invention include the time at which the 

statistics were computed such that those statistics are sent over multiple paths 
for improved reliability and to take advantage of one path having less delay than 
another. One embodiment at the receiving end is able to compare the 
computation times of received statistics to place them in their original temporal 

25 order, regardless of their relative arrival tinies over the paths. 

Some embodimejits of this invention will send computed statistics 
specific to the paths that are part of the plurality of one or more paths that are 
between the specific sender and receiver. Other embodiments will send 
additional computed statistics for paths that are not one of the plurality of one or 

30 more paths that are between the specific sender and receiver. 

Some embodiments of this invention will include network information 
concerning network topology including but not limited to information retrieved 
from routers such as in-bound or out-bound link utilization, inbound or out- 
bound link bandwidth and/or CPU utilization. Other network information 

35 determined from routers and other network devices wall be apparent to someone 
skilled in the art. 
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Some embodiments of this invention will also include control data 
directing a receiver of the measurement packet to change one or more 
configuration parameters of the receiver. 

In some embodiments of the invention, the control data will instruct a 
receiver to alter its configuration, including but not limited to zero or more of 
the following examples: instructing a receiver to initiate sending a pluralitv of 
one or more measurement packets, change one or more of the measurement 
packet sizes, inter-measurement packet transmission times and mix of packet 
sizes, and stop sending one or more of the plurality of measurement packets. 

In some embodiments of the invention, this control information will 
include notification of measurement devices that have joined or left the 
network. 

In many embodiments of the invention, the measurement packets will be 
encrypted by the sender and decrypted by the receiver. Some of these 
embodiments will use IPSec. 

In some embodiments of the invention, the encryption and decryption 
will be done by an external device using IPSec. 

Other encryption and decryption options will be apparent to one skilled 
in the art. 

In some embodiments of the invemion, the measurement packets will be 
digitally signed. 

In some embodiments of the invention, a generic communication 
channel will be used by a sender and a receiver to communicate data between 
them. 

Performance Characteristics of a Path 

Measurements are used to compute performance characteristics of the 
paths traversed by the measurement packets. The measurements can either be 
computed from the measurement packets themselves, or extracted from the 
arbitrary data carried by the measurement packets. The measurements of 
performance characteristics include at least one or more of one-way 
measurements and round-trip measurements. The performance characteristics 
.nclude at least one or more reachability, delay, jitter, loss, available bandwidth 
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and total bandwidth- Other performance characteristics will be apparent to those 
skilled in the art 

In some embodiments of the invention, delay measurements are 
computed as the inten'al of time from the moment the measurement packet is 
sent by the sender to the moment of time the measurement packet is received by 
the receiver. The sending time is carried by the packet, and it is measured by the 
clock the sender refers to. The receiving time is measured by a clock that the 
receiver refers to, which may or may not be synchronized with the sender's 
clock. 

In some embodiments of the invention, the clock of the sender and the 
clock of the receiver are synchronized. A plurality of one or more precise clock 
inputs such as GPS, NTP, IRIG and NIST will be used. Some embodiments of 
this invention will use the same clock as an input to more than one of the 
plurality of one or more senders and receivers. In some embodiments of the 
invention, the clock of the sender and the clock of the receiver are the same. 

In some embodiments of the invention, the clock of the sender and the 
clock of the receiver are not synchronized, and mechanisms based on the 
measurement data are used to coirect the clock skev^^ and clock drift, the 
mechanisms including using minimum delay across multiple measurement 
samples, and using a mechanism to track the minimum delay over time. 

Some embodiments of the invention will use the minimum round-trip 
delay between the devices to place a lower bound on clock skew. 

Some embodiments of the invention will use the lower bound of 
multiple paths between the sender and receiver toTurther reduce the lower 
5 bound. 

Some embodiments of tlie invention will correct for clock drift by 
tracking the relative clock skew between the sender and receiver over time and 

adjusting for the slope of the drift. 

hi some embodiments of the invention, jitter measurements, also known 
as inter-measurement packet delay variations, are computed as the difference in 
delay on consecutive, successfully received packets. 
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[ as 
' average 


In some embodiments of the invention, jitter can also be computed 
the dtfference between tl,e instantaneous delay of a packet, and the , 
delay of all the measurement packets previously received. 

In some embodiments of the invention, loss measurements are computed 
by asstgning a timeout value to each measurement packet that indicates the 
■nstan, of time after v,hich ,l,e measurement packet will be declared lost if ,l,e 
packet has no. arrived by that time. ,„ some embodiments of the inventi™ .be 
..meou. value of a measuremen. packe. can be comp„.ed with the ..nsmislion 

of a prevtously received packet, an estimation of the inter-transmission 
..me between measurement packet, and an estimation of ,be transmission delay 
of the measuremen, packet. In some embodiment of.be inven.ion. the inter- ' 
ransmission time can be cs.ima.ed if ,be receiver ^ows abou. .he scheduling 
pattern of ,r.,smissio„ of measuremen, packe.s. ,„ some embodimen.s of.!,; 
.nven«„„, d,e ,ra„smission delay of packe, can be esUma.ed based on delav and 
Jitter performance characteristics. 

Perfon,.ance charac,eris.ics of a path could be .he measuremen. 
emseives, or staristics on .hose measurements. ,„ .he s.a.is,ics case, a dynamic 
"-gor,.l™ rs used .o upda.es .he s,a.is,ics associated wi* a path with 


every new 


n.=asu,.me„t obtained with the arrival of eveo- new packet over the path 

In some embodiments of the invention, ,be algori.l™ computes statistics 
over the perfom,a„ce characteristics of tire path. 

m some embodimen,s of the inven.ion. .he s,a.is.ics include aver. 


•ages, 


devMons. and variances. Cher s.a.is.ics w.ll be apparen. .0 those skilled in the 
a«. ^ some embodiment of the invention, averages can be computed us „ 
Plural.ty of one or more techniques including a movi„. avenge 
--n.he.obbins-Moroes.ima.or,awi;dow.ba:::Z~^^^ 

In some embodimems of.he inven.ion. .he moving average is 
«ponen.,a,ly moviug average compu.ed using a Robbias loro Is il l- t, 
.ob.„s..oro3.o. 
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E[f(t)'^] = 0 

where E is the expectation, f(t) a function and x the estimator. The general form 
of the solution is: 

^(t) = + alpha * [ f(l) - x(i - 1) ] ^ (1 - alpha) * xO-1) + alpha f(() 
5 or, with alpha (} - a), 

x = u *x-^(J -u) V 

u is the weight of the estimator, and determines the amount contributed to the 
average by the function.. In some embodiments of the invention, p is constant. 
1 0 In some embodiments of the invention, p is a dynamic value, whose value 
depends on the last value of the function f according to the formula: 

p^e-(.f/K) 

where K is a constant that also determines the importance of the last value of/ 
with respect to the current value of the estimator a*. 
1 5 In some embodiments of the invention, average delay can be computed using an 
exponentially moving average as follows, 

= // * + ("7 - //; * w 

20 where d is the exponentially moving average of delay, m is the last delay 
sample, and p is the weight of the moving average. 

In some embodiments of the invention, average jitter can be computed using an 
exponentially moving average as follow^s, 

v = // *v + (1 ~ p) ^\d-m\ 

where v is the exponentially moving average of jitter, \d - m\ is the last sample 
oFjitter, and // is the weight of the average. 

30 
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In some embodiments of ,he mvenlion, aven,g. ji«er can be computed „sin» an 
exponenlially movine average as foljows, 

Where v is ,ha expone„>ia/,y m„vi„g average of ji„e, |„, - ,„ , i, ,„e 

sample of ji„er. „, is ,hc las, delay sample. ,„ ■ is the prev.ous delay sample, and 

// IS the weight of the averaoe 

In so..e embodiments of the invention, delay and jitter avera.es can be 
combined mto a single value as follows: 


Whe,. rfis ,he average delay, v is .he average jitter and Mi. a constant. 

m some embodiments of the invention, overage loss can be computed usin. 
exponentially moving average as follows. 

Where p-ha. is the tnoving average of the loss, p - (0 if packet is received , 
the pacta is declared lost,, and , is the weight of the exponentially m, " 


an 


1 IS 

o 


notio '7""™^'"""'="'^ .nvention.,,, determined based on the 
no o„ of fo,g,ve„ess against a single packet loss. n,e forgiveness period is the 
nerval of „me between the time the packet loss occu,. and the time the 
-ag. loss ,s forgiven. The forg.veness period can be either defined in u.ts of 

. . or ,n number Of packets if the rate of the monitoring «o„ ,s known. Th^ 
■ y^"^' ^'"^ " consecutive packets have been 

-,ved after the loss, when these packets have been transmitted at a certa.n 

The value of the exponentially moving avetage after receiving the n 
packets IS needed before u can h . ■ . 

// can be determmed, and this value is known as the 
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forgiveness threshold. In some embodiments of the invention, the forgiveness 
threshold is chosen arbitrarily. In some embodiments of the invention, the 
forgiveness threshold takes the value: 

5 This value is half of the value of the estimator after the singe loss occurs, and 
thus we call it the half-life Ihreshold. Similarly, we also call the forgiveness 
period under this threshold the half life period. The advantage of using a 
forgiveness threshold greater than zero is that issues related to host-dependent 
floating-point representations reaching that value are avoided. 

1 0 In some embodiments of the invention, // is computed by comparing the value 
of the estimator after n consecutive packet arrivals since the loss with the half- 
life Ihreshold: 

p^hat = (1- fO''^^n<y2(l - fi) 

Given that n is known because is determined by the value of the half life period 
\ 5 and the transmission rate, // is computed as: 

H =exp((In V2)/n) 

some embodiments of the invention, two thresholds are defined, an upper 
threshold and a lower threshold. When the value of p-hat exceeds the upper 
threshold, the loss is not forgiven until enough measurement packets are 
20 received consecutively so that the value of p-hat gets below the lower threshold. 

Other mechanisms to compute // will be apparent to for those skilled in the art. 

Path Description 

In some embodinients of the invention, the path traversed by the 
25 measurement packets from the sender to the receiver is such that the path is at 
least partly implemented with at least one of a GRE tunnel, an IPSEC tunnel 
and IPonIP tunnel. Other path implementations using tunnel will be apparent for 
those skilled in the art. 
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In some embodiments of the invention, the path traversed by the 
measurement packets from the sender to the receiver is implemented with a 
virtual circuit, including a frame relay PVC, an ATM PVC or MPLS. Other path 
implementations using virtual circuits will be apparent for those skilled in the 
art. 

Other path implementations will be apparent to those skilled in the art. 
Internetwork Description 

In some embodiments of the invention, the internetwork is implemented 
by a plurality of one or more subnetworks, including a plurality of one or more 
VPNs, a plurality of one or more BGP autonomous systems, a plurality of one 
or more local area nehvorks, a plurality of one or metropolitan area netvs'orks, 
and a pluralit\' of one or morewide area networks. 

In some embodiments of the invention, the internetwork is implemented 
by an overlay network. 

Other internetwork implementations will be apparent to those skilled in 

the art. 

Packet Sizes and Transmission Times 

In some embodiments of the invention, the measurement packets are of 
varying sizes, including 64, 256, 512, 1024, 1500 bytes. 

In some embodiments of the invention, the size of the measurement 
packets is specified with an external API. 

In some embodiments of the invention, the measurement packets are of a 
fixed size. 

In some embodiments of the invention, the measurement packet sizes 
and times between measurement packets simulate the traffic pattern of a 
plurality of one or more applications 

In some embodiments of the invention, traffic patterns correspond to voice 
applications, where the packets re of small size, e.g., 30 bytes, and the inter- 
transmission time between consecutive packets is constant, e.^.. 10 ms These 
examples do not limit the possible size values and inter-transmission lime 
values. 
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In some embodiments of the invention, traffic patterns con*espond to 
video applications, where the packets size is the largest permitted to be 
transmitted by an internetwork without being fragmented, and the inter- 
transmission time between consecutive packets varies depending on the spatial 
5 and temporal complexity of the video content being transmitted, the 
compression scheme, the encoding control scheme. 

hi some embodiments of the invention, traffic patterns correspond to the 
plurality of applications observed in an internetwork, including at least one or 
more of HTTP transactions, FTP downloads, IRC communications, NNTP 
10 exchanges, streaming video sessions, VoIP sessions, videoconferencing sessions 
and e-commerce transactions. Other types of applications will be apparent to 
those skilled in the art. 

In some embodiments of the invention, the inter-measurement packet 
transmission times are of varying length. 
1 5 In some embodiments of the invention, the inter-measurement packet 

transmission times are of fixed length. 

In some embodiments of the invention, the inter-measurement packet 
transmission times specified with an external API. 

In some embodiments of the invention, the length of the inter- 
20 measurement packet transmission times is randomized according to a 

distribution. In some embodiments of the invention, this distribution is based at 
least in part on a uniform distribution. In some embodiments of the invention, 
this distribution is based at least in part on an exponential distribution. In some 
embodiments of the invention, this distribution is based at least in part on a 
25 geometric distribution. Other distributions will be apparent to those skilled in 
the art. 

In some embodiments of the invention, the length of the inter- 
measurement packet transmission times is provided by a table. 

In some embodiments of the invention, the length of the inter- 
30 measurement packet transmission times is controlled by a scheduler. In some 
embodiments of the invention, the scheduler uses a priority queue, keyed on 
desired send time. 
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Other mechanisms to specify the inter-measurement packet transmission 
time will be apparent to those skilled in the art. 

Other packet sizes and transmission times will be apparent to those 
skilled in the art. ' 
Path Selection 

It is possible that multiple alternative paths between a sender and a 
receiver are available through at, internetwork at any given moment. 
Performance characteristics of each of these paths can be used to select a subset 
of the paths. 

In some embodiments of the invention., the subset of the plurality of 
paths is selected based at least in part on at least one of: one or more of 'the 
measurement statistics from the measurement packet and one or more of the 
computed statistics. 

In some embodimenG of ,h= invention, tte selection of the subset of the 
plurality Of paths is based at least partly „„ ,he position „f p„h, , „^,„, ,„ 
so.e embodiments of tlte invention, the ™king is at least pa„lv based on'one 
or more of the measnrement statistics included as data in the measurement 
packet. In some embodiments of the invention the ranking is at least patlly 
based on the computed statistics of the path. ,„ some embcdlmems of the 
-ention the ranking is implemented by using a comparison function to 
->pare the p..,.. ^ „,,,„^ ^ ^^^^ ^ ^^^^^ 

cmbod-ments of the invention the m,*ing is implemented by usi„» a 
coa,pa„son function ,„ compare 1,e paths, and by ordering the paths in'an 
■n-asmg order. Other ranking techniques ,v,l, be apparent to those skiiled in 

m some embodiments of the invenfon. the ranking is based on a single 
score assocated to each pa„. ,„ some embodiments of the invention, this score 
IS denoted Mag/c Score (MS), and it is computed as follows. 

MS = ML * AJF 
ML = d + PJ * v 
MF=clcllci*p-hai + I 
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where ML is the Magic Latency, a component of the MS obtained using delay 
and jitter respectively calculated with statistics; and MF is the Magic scaling 
Fac/or that multiplies the value of MU and is computed based on loss statistics. 
M is a constant that takes several values, including 4, for example. MS can be 

5 seen as a scaled-up version of ML, and the scaling factor MF is a function of p- 
hat and delta, a constant. As p-hat not only reflects loss but also detects large 
delay spikes before they happen, p-hat can be seen as an indicator of the 
departure of the path from a "normal mode" operation, and thus the scaling 
factor is only applied when there are loss or spikes. The goal of MF is to 

1 0 differentiate between paths that have verj' similar delay characteristics, but with 
one having losses and the other not having them. 

In some embodiments of the invention, ML is used as a delay indicator, 
given that jitter is accounted as an increase in delay. In contrast, MS, although a 
scaled version of ML, cannot be used to indicate delay, except when MF = 1 (p- 
15 hat = 0), which leads to MS = ML. That means the value of MS is useful not by 
itself but to compare it with the MSs of other tunnels. 

In some embodiments of the invention, loss statistics can be used as a 
■ discriminator instead of a scaling factor. That is, p-hat can eliminate paths 

experimenting loss. Then, the remaining paths can be selected using MS = ML. 

20 In some embodiments of the invention, the selection of a subset of paths 

is based on applying at least one or more thresholds to at least one of more of 
the statistics. 

In some embodiments of the invention, a single threshold is used, and 
computed as a certain percentage of the highest score of the paths. In some 
25 embodiments of the invention, the threshold is determined by subtracting a 
fixed quantity to the highest score of the paths. 

In some embodiments of the invention, the number of paths in the subset 
of paths is fixed. In some embodiments of the invention, this fixed number of 
paths N out of M paths is determined such that the probability of having loss in 
30 (M - N) paths simultaneously is less than a certain threshold. In some 
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embodiments of the invention, this probability is a binomial, with the 
assumption that ail paths have the same probability of loss. 

In some embodiments of the invention, the selection of the subset of the 
plurality of paths is based at least partly on a probability associated with each 
path. In some embodiments of the invention, the probability of each path is at 
least partly based on one or more of the measurement statistics included as data 
in the measurement packet. 

In some embodiments of the invention, the probabilities of each path are 

equal. 

In some embodiments of the invention, the selection of the subset of the 
plurality of paths is based at least partly on the cost of the path. 

In some embodiments of the invention, the selection of the subset of the 
Plurahty of paths is based at least partly on the amount of bandwidth consumed 
over a period of time. 

Other possibilities to compute path probabilities will be apparent to 

those skilled in the art. 

other mechanisms to select a subset of the paths will be apparent to 
those skilled in the art. 
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CLAIMS 

What is claimed is: 

1 . A method for characterizing a quality of a network path, including a first 
segment and a second segment, the method comprising: 

5 accessing a first metric and a second metric, 

wherein the first metric and the second metric are at least in part 
quality characterizations of a same plurality of one or more network 
applications, 

the quality characterization characterizes a quahty of the same 
1 0 plurality of one or more network applications running at one or more 

segment end-points, 

the first metric and the second metric are at least partly a 
function of a same plurality of one or more elementary network 
parameters. 

1 5 the plurality of one or more network parameters include one or 

more of delay Jitter^ loss, currently available bandwidth, and intrinsic 
bandwidth, 

the first metric is at least partly the function of the same plurality 
of elementary network parameters of the first segment, 
20 the one or more segment end points include one or more end- 

points of the first segment, 

the second metric is at least partly the function of the same 
plurality of elementary network parameters of the second segment, and 
the one or niore segment end points include one or more end- 
25 points of the second segment; and 

adding the first metric and the second metric to generate a third metric, 

wherein the third metric is at least partly the function of the same 
plurality oft' one or more elementar>' network parameters of the network 
path, 

30 the one or more segment end points include one or more end- 

points of the network path, and 
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the third metric is a quality characterization of the same plurality 
of one or more applications. 

2. The method of 1 , further comprising: 

prior to accessing the first or the second metric, generating at least one 
of the first metric and the second metric. 

3. The method of 1, further comprising: 

prior to accessing the first or the second metric, receiving at least one of 
the first metric and the second metric. 

4. The method of claim 1 , wherein at least one of the plurality of one or 
more network parameters is dynamic. 

5. The method of claim 1, wherein at least one of the plurality of one or 
more network parameters is static. 

6. The method of claim 1 , wherein the plui ality of one or more network 
applications include at least one of UDP and TCP applications. 

7. The method of claim 6, wherein the plurality of one or more network 
applications include UDP applications. 

8. The method of claim 7, wherein the plurality of one or more network 
applications include voice. 

9. The method of claim 7, wherein the plurality of one or more net\vork 
applications include video. 

1 0. The method of claim 9, wherein the plurality of one or more network 
applications include video conferencinc-, 

i 1. The method of claim 6, wherein the plurality of one or more network 
applications include TCP applications. 
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12. The method of claim 1 1 , wherein the plurality of one or more network 
applications include HTTP. 

13. The method of claim 1 2, wherein the plurality of one or more network 
5 applications include HTTP/1 .0. 

1 4. The method of claim 12, wherein the plurality of one or more network 
applications include HTTP/1 . 1 . 

10 15. The method of claim 1 1 , wherein the plurality of one or more network 
applications include ftp. 

1 6. The method of claim 1 1, wherein the plurality of one or more network 
applications include telnet. 

15 

1 7. The method of claim 1 , wherein the plurality of one or more network 
parameters include delay. 

1 8. The method of claim 1 , wherein the plurality of one or more network 
20 parameters include jitter. 

1 9. The method of claim 1 , wherein the plurality of one or more network 
parameters include loss. 

25 20. The method of claim 1 , wherein the plurality of one or more network 
parameters include currently available bandwidth. 

2 1 . The method of claim 1 , wherein the plurality of one or more network 
parameters include intrinsic bandwidth. ^ 

30 

22. The method of claim 1 , wherein the metric includes non-performance 
related characteristics. 
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23. The method of claim 21, wherein the non-performance related 
characteristics includes pre-specified route preferences. 

24. A network system, comprising: 

a plurality of one or more network devices configured, such that if the 
network device is coupled to at least a network path including a first segment 
and a second segment, the plurality of one or more network devices performing: 

accessing a first metric and a second metric, 

wherein the first metric and the second metric are at least in part 

quality characterizations of a same plurality of one or more network 

applications, 

the quality characterization characterizes a quality of the same 
plurality of one or more network applications running at one or more 
segment end-points, 

the first metric and the second metric are at least partly a 
function of a same plurality of one or more elementary network 
parameters, 

the plurality of one or more netAvork parameters include one or 
more of delay Jitter, loss, currently available bandwidth, and intrinsic 

bandwidth, 

the first metric is at least partly the function of the same plurality 
of elementar>' network parameters of the first segment, 

the one or more segment end points include one or more end- 
points of the first segment, 

the second metric is at least partly the function of the same 
plurality of elementary network parameters of the second segment, and 

the one or more segment end points include one or more end- 
pomts of the second segment; and 

adding the first metric and the second metric to generate a third metric 

wherein the third metric is at least partly the function of the same 
Plurahty of one or more elementary network parameters of the network 
path, 
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the one or more segment end points include one oi* more end- 
points of the network path, and 

the third metric is a quality characterization of the same plurality 
of one or more applications. 

5 

25. The network system of 24, wherein the network device further performs: 

prior to accessing the first or the second metric, generating at least one 
of the first metric and the second metric. 

10 26. The network system of 24, wherein the network device further performs: 
prior to accessing the first or the second metric, receiving at least one of the first 
metric and the second metric. 

27. The network system of 24, wherein at least one of the plurality of one or 
1 5 more network parameters is dynamic. 

28. The network system of 24, wherein at least one of the plurality of one or 
more network parameters is static. 

20 29. The netw^ork system of 24, wherein the plurality of one or more netw-ork 
applications include at least one of UDP and TCP applications. 

30. The network system of 29, wherein the plurality of one or more network 
applications include UDP applications. 

25 

3 1 . The network system of 30, wherein the plurality of one or more network 
applications include voice. 

32. The network system of 30, wherein the plurality of one orTuore network 
30 applications include video. 

33. The network system of 32, wherein the plurality of one or more network 
applications include video conferencing. 
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34. The network system of 29, wherein the plurality of one or more network 
applications include TCP applications. 

35. The network system of 34, wherein the plurality of one or more network 
applications include HTTP. 

36. The network system of 35, wherein the plurality of one or more net^vork 
applications include HTTP/1.0. 

37. The network system of 35, wherein Uie plurality of one or more network 
applications include HTTP/1.1. 

38. The net^vork system of 34, wherem the plurality of one or more network 
applications include ftp. 


39. The network system of 34. wherein the plurality of one or more network 
applications include telnet. 


40. The network system of 24, wherein the plurality of one or more network 
parameters include delay. 


41 . The nehvork system of 24, wherein the ph,rali,y of „„e or more nehvork 

parameters include jitter. 


12. The network system of 24, wherei,, the plurahly of o„e or more network 
parameters include loss. 


43. The network system of 24. wherein the plurality of one or more network 
parameters include currently available bandwidth. . 


44. The network system of 24, wherem the plurality of one or more network 
parameters include intrinsic bandwidth. 
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45 The network system of 24, wherein the metric includes non-performance 
related characteristics. 

46. The network system of claim 45, wherein the non-performance related 
5 characteristics includes pre-specified route preferences. 

47 . The network system of 24, ftjrther comprising: 

a plurality of one or more inputs adapted to be coupled to the network 

path; and 

1 0 a plurality of one or more outputs coupled to the plurality of one or more 

inputs, 

wherein responsive to a plurality of one or more packets an iving to the 
network device through the plurality of one or more inputs, the network device 
selects at least one output from the plurality of one or more outputs, and 
1 5 the at least one output is determined at least partly using at least one of 

the first metric, second metric, and third metric. 
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